Engineering Pornography 


“This is one of my favorite geek stories of aii time, i know i've mentioned it 
to a few of you recentiy, and you hadn't seen it, so here it is for posterity.” 

Jamie Zawinsk 24 November 2002 


Date: Wed, 27 Sep 89 10:27 PDT 

From: "Thomas L. Me Mahon" <tlm@riverside.scrc.symbolics.com> 
Subject: Cuts and jumpers (on a different scale) 

To: Hardware@riverside.scrc.symbolics.com 

[Not quite the right mailing list but close. If you don't care about megawatts, 
bus bars bigger than your wrist, things that cause ground loops out to 
Hawaii, or big hairy construction projects hit D now.] 


Several days ago a very large number of trucks and men from the Los 
Angeles Department of Water and Power descended on my neighborhood. 
They removed large sections of Pershing drive to a depth of 15 feet or so 
over a stretch of about a city block. I assumed they had a problem with a 
water main or something. 

When they started building semi-permanent structures over the holes I 
knew something really big was up. When the large trucks full of strange 
power tools, mega-welding machines, breathing equipment, and racks of 
test equipment came I started wondering. Driving by a couple nights ago 
(11 PM), I noticed that the pace hadn't slowed - they were at it 24 hours a 
day. 

My curiosity got the best of me yesterday when they brought in the giant 
tanks full of liquid nitrogen. LN-2 for the DWP? I parked my car and played 
the lookie loo. 



It turns out they have a problem with an underground wire. Not just any 
wire but a 230 KV, many-hundred-amp, 10 mile long coax cable. It shorted 
out. (Lotta watts!) It feeds (fed) power from the Scattergood Steam Plant in 
El Segundo to a distribution center near Bundy and S.M. Blvd. 

To complicate matters the cable consists of a copper center conductor 
living inside a 16 inch diameter pipe filled with a pressurized oil dielectric. 
Hundreds of thousands of gallons live in the entire length of pipe. Finding 
the fault was hard enough. But having found it they still have a serious 
problem. They can't afford to drain the whole pipeline - the old oil 
(contaminated by temporary storage) would have to be disposed of and 
replaced with new (pure) stuff which they claim takes months to order (in 
that volume). The cost of oil replacement would be gigantic given that it is 
special stuff. They also claimed the down time is costing the costing LA 
$13,000 per hour. How to fix it and fast? 

That's where the LN-2 comes in. An elegant solution if you ask me. They 
dig holes on both sides (20-30 feet each way) of the fault, wrap the pipe 
with giant (asbestos-looking) blankets filled with all kind of tubes and wires, 
feed LN-2 through the tubes, and *freeze* the oil. Viola! Programmable 
plugs! The faulty section is drained, sliced, the bad stuff removed, replaced, 
welded back together, topped off, and the plugs are thawed. I was amazed. 


The next day: 

Last night the DWP held a curbside chat to allay the neighborhood's fears 
that they were going to accidentally blow us all up. Apparently all the vapor 
clouds from all the LN-2 blowoff had caused a great deal of concern. 

Interesting bits: 

The feeder was laid 17 years ago and was designed to have an MTBF of 
60 years. There are other similar feeders in use around California, in the 
Pacific North West, and some on the east coast. This was the first failure in 
the western US. No one out here had any idea how to fix it so they brought 
in experts from the east. (NYC has had some faults.) 

This link is a very critical part of the LA power grid. Last night the city 
engineers verified the $13,000 per hour power cost figure quoted the day 




before. (I guess that means they are being forced to buy power off the grid 
somewhere else.) 

There are actually three center conductors (they had a cross sectional 
model to show us). Each is about 3" in diameter with a one inch solid 
copper core. Each is wrapped with hundreds of layers of a special paper. 
That, in turn, is sheathed with copper and then each one is spiral-wrapped 
its entire segment-length with a 1/4 inch bronze "wire". The three 
conductors are then twisted together during the pulling process. The 
bronze spiral wraps form a kind of linear bushing with minimal contact area 
with the inside of the pipe so it's "easy" to pull each segment. Ha. 

Each of the three legs in the feeder carrys 600-800 Amps (depending on 
demand) of 230KV three phase power. The ground return is the Santa 
Monica Bay. Down at the Scattergood Steam Plant and up in Santa Monica 
they have a giant copper anchors out in the bay. 

They lay these things in 2000 foot segments. 2000' is the longest segment 
they can pull through the steel pipe. The pipe is laid first and then the 
internal cable(s) are pulled through. Tensile forces must be enormous. At 
each segment joint (splice) there is a very large and expensive ($100K) 
underground vault. Future technology may allow them to go 3000 feet, 
reducing the number of vaults needed per run, thereby saving money. 


After the feeder was originally built (and the cable pulled) it was thoroughly 
evacuated to both leak test and remove any contaminants. It was flushed 
with dry nitrogen and then reevacuated. Golden Bear High Tension Oil was 
then slowly added while still maintaining a vacuum so as to "pull" any 
residual gas contaminants out of the oil and the cables in the pipe. The 
pipe, full of oil, is then pressurized to about 200 PSI for some period of time 
before it gets powered up. 200 PSI is maintained during operation to keep 
any bubbles from forming and to drive insulating oil into the paper. 

At both ends of the pipeline they have 6000 gallon tanks of Golden Bear 
lightly pressurized under a blanket of dry nitrogen. There are pumps at both 
ends. There is about 100K gallons in the entire pipeline, not including the 
6K gal tanks. Every six hours they reverse the pumps so the oil oscillates 
back and forth in the pipe. The pumps only run at 3 gallons per minute but 



that is enough, over 6 hours, to get the oil in each 2000 foot segment to go 
at least a segment or two length in either direction. This eliminates hot 
spots in the copper conductors and spreads the heat out over several 
thousand feet. A little competitive pressure is always maintained between 
the pumps to get the 200 PSI. 

They learned the hard way that you simply don't reverse the pumps lest 
you get the Golden Bear equivalent of water hammer. The last hour of 
every 6 hour cycle is spent slowly reducing the oil velocity down to zero 
before you reverse it and then slowly ramp back up in the other direction. 

In between segments, in the vaults, are temperature sensors embedded in 
the pipe. These monitor the oil temperature. These are wired to a computer 
downtown. Because the oil oscillates, the DWP can track the temperature 
gradient along the pipe and get an early indication of the location of any hot 
spot problems. They have regularly spaced flow rate and pressure monitors 
for the same purpose - detecting and isolating faults. 

Every vault also has a nipple which allows sampling of the pipe oil. They 
said you withdraw the oil through a thick membrane with a syringe (?). This 
happens monthly on all feeders in the LA area. The samples are analyzed 
downtown by a staff of chemists who can relate the presence of things like 
acetylene, butane, and benzene in the oil to arcing, coronas, and so forth. 
Apparently the oil chemistry is a very good indicator of the health of the 
segments. 

One of their worst fears, after they open up the pipe, is having a blowout of 
the freeze plugs. If they ever run out of nitrogen during the repair process 
they'll lose one side of the pipe (or both). Right now they've got the pipe on 
each side of the fault dropped down to 80 PSI. They are afraid that if they 
go any lower in oil pressure any gas in the oil will come out of solution and 
cause an explosive expansion. Not only that, but since there is so much oil 
embedded in the paper insulation, any sort of gas bubbling (oil foaming) 
would shred the insulation, rendering the entire feeder useless. They say it 
could take months to safely let the pressure off to zero. (That is the other 
reason ($13k/hr) they cannot afford to drain the whole pipe.) 

Even at 80 PSI, if they lose a freeze plug they will have a really big mess 
outside the pipeline. The holes they've dug cannot hold 10OK gallons and 
they're operating on a hill near the beach anyway... (Big pollution threat for 



LA basin.) Potentially fatal for anyone around. Right now they have LN-2 
companies on call from San Diego to San Francisco with contingency plans 
of all sorts in case there is a major traffic problem with trucks getting in. 

They say the repair could take weeks or more, depending on what they find 
when they get inside. They believe the cause of the fault was the inner 
conductors slipping downhill inside the pipe and shorting against a metal 
flange. Even if that's true they wonder where it slipped to, and hence, 
where it may be bunched up down hill. 

Finding the fault was a problem in itself. Since this was all new to them they 
really didn't know how to start. They tried time-domain reflectometry 
equipment but got inconclusive information. They tried ultrasound and 
radar but that didn't work. Then they got a thing called a "thumper" shipped 
in which got them pretty close. The thumper sends mondo-amp pulses into 
one end of the cable. The electromotive force tends to cause physical 
displacement of the conductors which you can hear from the street level. 
The place where the clicking stops is where the problem usually is. This got 
them to the defective segment. 

What pinpointed the problem in the end was a bunch of car batteries and 
some millivoltmeters. (From one technology extreme to the other.) They 
hooked up car batteries to both ends, tapped the cable at several points 
(maybe there are taps in the vaults?) and, knowing the drops and 
resistance of the cable, got within a few feet of the fault. (I used to use the 
exact same technique on memory boards.) 

Next came the X-Ray equipment. Sure enough, they can see the cable 
shorting against the steel wall of the pipe. 

Once all of the repair is done they still have to close it up. How do you weld 
a steel pipe with paper insulation inside? Slowly. They have special heliarc 
welding equipment and "certified operators" who take 8 hours to weld 
around one cross section of pipe. They are required to keep their hand on 
the pipe no more than 3 inches from the tip of the welder. If it is too hot for 
their hand they stop and let it cool. After all, they can't afford another 
failure. 


Oct 20, 1989 Update 




I am getting all of these bits from a guy named Jim who is the project 
manager. He looks kinda like a red neck RWK (Jesus in a hard hat with a 
Harley belt buckle). [RWK is Bob Kerns, an ex-Symbolics person, 67" tall, 
skinny, bearded. -- DLW] He is a really great guy. Jim was one of the 
splicers on the project 17 years ago when he was working his way through 
school. He is a now professional electrical and mechanical engineer. After 
having worked his way up through the ranks at the DWP he is now The Big 
Boss. He claims to be having the time of his life - back in the field with one 
of the biggest challenges of his career. If we ever recruit a VP of 
engineering I would hope its someone like him. 

So, what went wrong? Varying load conditions in the three legs of the 3- 
phase circuit caused tremendously strong and dynamic magnetic field 
changes. The electromagnetic forces between the three conductors and 
the steel pipe (gack!) cause the conduit to wiggle around inside the pipe. 
Over many years (and under the influence of gravity) the thing slipped and 
wiggled every which way. Also, due to very slight diametric temperature 
gradients, the differential thermal expansion of a cable that big across 
causes bending and warping forces. Nobody ever thought of any of this. 

Wiggle alone may not have cause the problem, however. The spices 
between cable segments are much larger in diameter than the cable itself. 
The steel pipe at these points is much larger than the main run. So the 
whole affair get fat and then shrinks down every 2000 feet or so. What 
really screwed them was failure to put any sort of clamp at the splices to 
keep the fat splice from getting pulled into the narrower main runs. This is 
what cause the fault. 

Jim says the fault lasted 20 milliseconds before breakers tripped. (The 
breakers for a wire like this are pretty amazing in their own right. They use 
high pressure gas to blow out the arc as the circuit begins to open. 

Anything that can cut off this number of megawatts in 20 ms gets my 
respect.) It blew carbonized oil about 3000 feet down the pipe to either side 
of the fault. (Compute velocity...) 

They will be removing a long length of cable from the faulted area for 
analysis. The entire length will be dissected. Jim says the insulation they 
have inspected at so far looks like shredded cauliflower due to the 



explosion from the fault and the gas bubbling in it. (BTW - The insulation 
consists of 118 layers of paper tape.) 

Based on X ray imaging they are going to have to open up 14 of the 23 
splices along the 10 mile run. They'll have to drain the pipe to do so. It will 
take them 2 months to take the pipeline down (depressurize and drain). 
(The oil will be recycled - see below.) At each of the opened splices they 
are going to install special aluminum (non magnetic) collars around the 
conductors to keep the splices from getting pulled into the necked-down 
section of the pipe. These collars are being specially fabricated now and 
will be ready in about a month. 

At each splice they have to build a semi clean room to keep dirt, moisture, 
worker sweat, and any other contaminants out of the joint before closing. 
After all, we're talking a quarter megavolt! They have special air 
conditioning and filter units for the vaults. Each joint will take two months of 
work. They will get some degree of parallelism in the phase of the project. 

After repairing and replacing the faulted section of cable, stabilizing all of 
the splices, and buttoning it all up comes the job of putting the oil back in. 
First the pipe is evacuated and then back-filled with nitrogen etc as I 
described earlier. After extensive filtering, the oil is heated to about 230 
degrees farenheit. It gets injected into a vacuum chamber at the 
temperature thorough hundreds of spray nozzles. This gets the maximum 
possible surface area so all the crap in it boils out into the chamber. The 
good stuff that's left is collected and pumped immediately into one end of 
the pipeline. 

Then they power it up and see if it works. If not, they start over again. I'll 
keep you posted. 


12/11/89 

The pipeline is now completely drained and filled with nitrogen at 
atmospheric pressure. They are still in the process of opening up many of 
the splices to install the collars. They originally planned on doing 14 of 
them but now the number is 17. 



The splice where the short happened, and the cable for about 40 feet to 
either side of it, has been completely replaced. Engineering prototypes of 
the aluminum collars are now installed there and the whole thing is all 
welded back together. They built new vaults under the street where the 
new splices are. The trucks, men, and their equipment are all gone and the 
roadbed has been repaved. They consider the segment where the short 
happened "fixed". We'll see... 

Now that they verified the assembly of a splice with the new collars in it (to 
make sure they had the details right) they are going ahead with the fab run 
for the 17 other joints. They won't be done installing them until mid April. 
Then they'll put the oil in as described above and power it up slowly. I don't 
expect to have much to report until then and I promise to send an update. 

Assuming 8 month down time the cost of the electricity alone will approach 
$75 million! 


[after many queries of "So how did it all work out?"] 

... I am sure that you will all get this final message in due time. 

You see, I found out yesterday that even the Department of Water and 
Power repair crew got copies! It rattled across electronic bulletin boards 
around the country and eventually ended up in the hands of a DWP 
manager who promptly showed it to the team. Small world. Every time you 
turn over a rock you find that we are all connected. 

So, here's the scoop: 

They are back on line. The failure was attributed to "TMB", short for 
Thermal Mechanical Bending. There have been several similar failures on 
the east coast but this was the first out here. TMB causes the cable to 
wiggle in place due to load surges. This eventually causes insulation failure 
due to abrasion against the pipe and separation of the many layers of 
paper tape. They repaired the short, put aluminum collars in most of the 
joints to hold the splices in place, and have added a load management 
scheme to reduce the current peaks. They powered it up and the darned 
thing works. Amazing. 

I'll impulse them again in a few months to see if there is any news. 





